Video compression is used in many current and emerging products. It has found applications in video-conferencing, video streaming, serial storage media, high definition television (HDTV), and broadcast television. These applications benefit from video compression in the fact that they may require less storage space for archived video information, less bandwidth for the transmission of the video information from one point to another, or a combination of both.
Over the years, several standards for video compression have emerged; such as the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) recommended video-coding standards: H.261, H.262, H.263 and the emerging H.264 standard and the International Standardization Organization and International Electrotechnical Commission (ISO/IEC) recommended standards MPEG-1, MPEG-2 and MPEG-4. These standards allow interoperability between systems designed by different manufacturers.
Video is composed of a stream of individual pictures (or frames) made up of discrete areas known as picture elements or pixels. The pixels are organised into lines for display on a CRT or the like. Each pixel is represented as a set of values corresponding to the intensity levels of the luminance and chrominance components of a particular area of the picture. Compression is based mainly on the recognition that much of the information in one frame is present in the next frame and, therefore, by providing a signal based on the changes from frame to frame a much reduced bandwidth is required. For the purpose of efficient coding of video, the pictures or frames can be partitioned into individual blocks of 16 by 16 luminance pixels called “macroblocks”. This practice simplifies the processing which needs to be done at each stage of the algorithm by an encoder or decoded. To encode a macroblock (or sub-macroblock partition) using motion-compensated prediction, an estimation is made of the amount of motion that is present in the block relative to the decoded pixel data in one or more reference frames, usually recently decoded frames, and the appropriate manner in which to convey the information from which the current frame may be reconstructed. The residual signal, which is the difference between the original pixel data for the macroblock and its prediction, is spatially transformed and the resulting transform coefficients are quantized before being entropy coded. The basic processing blocks of an encoder are a motion estimator/compensator/predictor, a transform, a quantizer and an entropy coder. Due to the quantization of the transformed coefficients of the residual signal, the reconstructed pixel values are generally not identical to those of the original frame. Since the coding is block-based, the errors that are introduced by the quantization process tend to produce artifacts in the form of sharp transitions in image intensity across transform block boundaries in the reconstructed frame. Such artifacts are referred to as “blocking artifacts”. The appearance of blocking significantly affects the natural smoothness seen in video images and leads to a degradation of the overall video image quality.
Blocking artifacts are inherent in hybrid block-based video coders, especially in low bit rate video applications. A number of solutions have been presented to alleviate the degradation in visual quality due to the presence of blocking artifacts. Two general approaches have been proposed to deal with blocking artifacts. The first approach is based on using a deblocking filter in the decoder only as a post-processing stage, and applying the deblocking filter on the decoded and reconstructed video frames before they are displayed. The purpose of the filter is to modify the sample values around the block boundaries in order to smooth unnatural sharp transitions that have been introduced by the block-based coding process. Having a deblocking filter applied outside of the motion-compensation loop can be viewed as an optional process for the decoder, placing no requirements on the video encoder. However, this scheme has a disadvantage in that the reference frames that are used for generating predictions for the coding of subsequent frames will contain blocking artifacts. This can lead to reduced coding efficiency and degraded visual quality. The second approach to reduce the visibility of blocking artifacts is to apply a deblocking filter inside the motion-compensation loop. In this case, the reference frames that are used for generating predictions for subsequent encoded frames represent filtered reconstructed frames, generally providing improved predictions and improved compression and visual quality. In order to create identical predictions at both the encoder and decoder, the deblocking filter (sometimes referred to as a “loop filter” if it is inside the motion-compensation loop) must be applied in both the encoder and the decoder.
In order to reduce the appearance of blocking artifacts, a number of video coding standards, including H.263 version 2, and most recently the emerging H.264 video coding standard specify the use of a deblocking filter inside the motion-compensation loop. In particular, the H.264 video coding standard fully specifies a deblocking filter that is to be used inside the motion-compensation loop in both the encoder and decoder.
One of the known prior art methods is described in a document “Working Draft Number 2, Revision 2 (WD-2)” by the Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG. In this prior art method, filtering occurs on the edges of 4×4 blocks in both the luminance and chrominance components of each reconstructed video frame. The filtering takes place on one 16×16 macroblock at a time, with macroblocks processed in raster-scan order throughout the frame. Within each macroblock, vertical edges are filtered first from left to right, followed by filtering of the horizontal edges, from top to bottom. The filtering of samples for one line-based filtering operation occurs along the boundary separating unfiltered samples p0, p1, p2, and p3 on one side of the boundary, and unfiltered samples q0, q1, q2, and q3 on the other side, as illustrated in FIG. 3a. The block boundary lies between samples p0 and q0. In some cases p1, p2 may indicate samples that have been modified by filtering of a previous block edge. For each line-based filtering operation, unfiltered samples will be referred to with lower-case letters, and filtered samples with upper-case letters. For each block boundary segment (consisting of 4 rows or columns of samples), a “Boundary strength” parameter, referred to as “Bs”, is computed before filtering. The calculation of Bs is based on parameters that are used in encoding the bounding blocks of each segment. Each segment is assigned a Bs value from zero to four, with a value of zero indicating that no filtering will take place, and a value of 4 indicating that the strongest filtering mode will be used.
The process for determining Bs is as follows. For each boundary, a determination is made as to whether either one of the two blocks that neighbour the boundary is intra-coded. If either block is intra-coded, then a further determination is made as to whether the block boundary is also a macroblock boundary. If the block boundary is also a macroblock boundary, then Bs=4, else Bs=3.
Otherwise, if neither block is intra-coded then a further determination is made as to whether either block contains non-zero transform coefficients. If either block contains non-zero coefficients then Bs=2; otherwise if a prediction of the two blocks is formed using different reference frames or a different number of frames and if a pair of motion vectors from the two blocks reference the same frame and either component of this pair has a difference of more than one sample, then Bs=1; else Bs=0, in which case no filtering is performed on this boundary. The value of boundary strength, Bs, for a specific block boundary is determined by the encoding characteristics of the two 4×4 blocks along the boundary. Therefore, the control of the filtering process for each individual block boundary is well localized. The block boundary is filtered only when it is necessary, based on whether the coding modes used for the neighbouring blocks are likely to produce a visible blocking artifact.
The known filtering process starts with the step of filtering each 4×4 block edge in a reconstructed macroblock. The filtering “Boundary strength” parameter, Bs, is computed and assigned based on the coding parameters used for luma. Block boundaries of chroma blocks correspond to block boundaries of luma blocks, therefore, the corresponding Bs for luma is also used for chroma boundaries.
Filtering takes place in the order described above on all boundary segments with non-zero value for Bs. The following describes the process that takes place for each line-based filtering operation.
TABLE 1QPav dependent activity threshold parameters α and βQPav0123456789101112131415161718192021222324252627α000000000000000000004456791012β0000000000000000000033344466QPav282930313233343536373839404142434445464748495051α141720242833394655657690106126148175207245255255255255255255β778899101011111212131314141515161617171818
A content activity check is performed. If the check is passed, filtering continues; otherwise, the sample values are not modified on this line of the boundary segment. The activity check makes use of a pair of activity threshold parameters, ALPHA (α) and BETA (β), whose particular values are selected from the above Table 1, based on the average quantization parameter (QPav) used in coding each boundary segment. It is noted that QPav represents the average value of the quantization parameter values used in encoding the two blocks that neighbour the boundary, with rounding of the average by truncation of any fractional part. Accordingly, the content activity check is passed if|p0−q0|<ALPHA(α) AND |p1−p0|<BETA(β) AND |q1−q0<BETA(β).
Further, if this first content activity check is passed, and Bs is not equal to 4, default mode filtering is performed. Otherwise, if the check is passed and Bs is equal to 4, a second, stricter activity check is performed. This activity check involves the evaluation of the condition1<|p0−q0|<(QPav>>2) AND |p2−p0|<BETA(β) AND |q2−q0|<BETA(β).If this second condition is true on a particular line of samples, a strong mode filtering is used on this line of samples. Otherwise, a default mode filtering is used on this line of samples. It should be noted the symbol “>>” is used to represent the operation of bit-wise shifting to the right.
Among the disadvantages of the above described known method is that it permits switching between two filtering modes with very different characteristics at the level of each line of samples within a boundary segment. This switching adds complexity to the filtering process and can significantly increase the worst-case critical path for processing on many architectures.
Further disadvantages include the particular values in the tables of filtering parameters, ALPHA (α) and BETA (β), which are not optimized to produce the best subjective viewing quality of reconstructed and filtered video. Further, the characteristics of the deblocking filter in terms of the threshold parameters used in the activity checks and equations used for generating filtered sample values are fixed in the known method, providing the encoder with little or no flexibility to control the properties of the deblocking filter. This hinders optimization of the subjective quality of the decoded video for different types of video content and displays.
In the default mode of the above identified filtering method, the value Δ, which represents the change from the unfiltered values of p0 and q0 to their respective filtered values is computed using:Δ=Clip(−C,C,(((q0−p0)<<2+(p1−q1)+4)>>3)),where C is determined as specified below and the function “Clip” is defined as:Clip(a,b,c)=IF(c<a)THEN a ELSE IF(c>b) THEN b ELSE c Further, the filtered values P0 and Q0 are computed whereP0=Clip(0,255,p0+Δ) and Q0=Clip(0,255,q0−Δ).
In order to compute the clipping value, C, that is used to determine Δ, and also determine whether the values of p1 and q1 will be modified on this set of samples, two intermediate variables, ap and aq are computed, where:ap=|p2−p0| and aq=|q2−q0|.
If ap<β for a luminance edge, a filtered sample P1 is produced as specified by:P1=p1+Clip(−C0,C0,(p2+P0−(p1<<1))>>1).
If aq<β for a luminance edge, a filtered sample Q1 is produced as specified by Q1=q1+Clip(−C0,C0,(q2+Q0−(q1<<1))>>1) where C0 is specified in Table 2 (see below), based on Bs and QPav for the block boundary. For both luma and chroma, C is determined by setting it equal to C0 and then incrementing it by one if ap<β, and again by one if aq<β.
TABLE 2Value of filter clipping parameter C0 as a function of QPav and BsQPav012345678910111213141516171819202122232425Bs = 100000000000000000000000111Bs = 200000000000000000000011111Bs = 300000000000000000111111111QPav2627282930313233343536373839404142434445464748495051Bs = 111111112222333444566789101113Bs = 211111222233344556788101112131517Bs = 312222333444566789101113141618202325
It is important to note that the computation of the filtered values P1 and Q1 require as an input to the filtering equation the filtered values of P0 and Q0 from the current line of samples. This recursive filtering method presents a disadvantage as the values of P0 and Q0 must be computed before the computation of P1 and Q1 can begin. This design can impede parallel processing of the different samples and thereby increases the critical path for the default mode filtering on most hardware architectures.
An additional disadvantage in the default mode filtering process of the known method is that the calculation of the clipping parameter, C, for chroma samples is unnecessarily complex. The chroma samples p1 and q1 are never filtered in the default mode and, therefore, the computation of the variables ap and aq is only necessary to determine the C parameter that is used to clip the value of Δ. These computations could be avoided by specifying a simpler method to compute C for chroma filtering.
For strong mode filtering in the known method, the following equations are applied to calculate the filtered sample values:P0=(p2+2*p1+2*p0+2*q0+q1+4)>>3,P1=(p3+2*p2+2*p1+2*p0+q0+4)>>3,Q0=(p1+2*p0+2*q0+2*q1+q2+4)>>3 andQ1=(p0+2*q0+2*q1+2*q2+q3+4)>>3.For the luminance component only, p2 and q2 are also filtered as specified by:P2=(2*p3+3*p2+p1+p0+q2+4)>>3 andQ2=(2*q3+3*q2+q1+q0+p0+4)>>3.
Filtering with this set of equations can lead to insufficient reduction in the visibility of blocking artifacts. It is therefore an object of the present invention to obviate or mitigate the above-mentioned disadvantages.